It's working okay now all things considered. I'm surprised at how well the scaling works as on the raspbian + retropie the text is harder to see and the borders are a bit off even in the terminal. I should try to copy retropie's font to my other sd cards with raspbian + retropie on them to make the console more visible

Glad you got it sorted out, actually RetroPie is using Raspbian (RPI OS) Lite underneath, so there's no specific font that's included in RetroPie.

Maybe the borders' position have something to do with the Overscan settings for the display ? Or you may try to increase the font size from the Setup script, as explained here ?